System for a Spectator to View a Virtual Reality of a Performer and Method

ABSTRACT

A system for a spectator to view a virtual reality of a performer includes a head mounted display worn by the performer which produces the virtual reality for the performer to see while wearing the head mounted display. The system includes an audience portion having an audience display for an audience member which receives input signals from a network corresponding to the virtual reality being viewed by the performer on the head mounted display and displays these input signals on the audience display for the spectator to view. A method for a spectator to view a virtual reality of a performer.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a nonprovisional of U.S. provisional application Ser. No.62/469,921 filed Mar. 10, 2017, incorporated by reference herein.

FIELD OF THE INVENTION

The present invention is related to a spectator using a display of amobile phone to view a virtual reality on a head mounted display worn bya performer. (As used herein, references to the “present invention” or“invention” relate to exemplary embodiments and not necessarily to everyembodiment encompassed by the appended claims.) More specifically, thepresent invention is related to a spectator using a display of a mobilephone to view a virtual reality on a head mounted display worn by aperformer wherein the virtual reality displayed on the audience displayof the smart phone corresponds to an actual position and orientation ofthe spectator in a room in which the performer having the head mounteddisplay and the spectator are located.

BACKGROUND OF THE INVENTION

This section is intended to introduce the reader to various aspects ofthe art that may be related to various aspects of the present invention.The following discussion is intended to provide information tofacilitate a better understanding of the present invention. Accordingly,it should be understood that statements in the following discussion areto be read in this light, and not as admissions of prior art.

In their current stage, virtual reality experiences can be veryisolating. An audience member present in a room who is watching someoneelse wearing a Head Mounted Display (HMD) can have very littleinteraction with the person wearing the virtual reality HMD. Bycombining each audience member's mobile phone with the virtual realityHMD via a local network, the current invention enables a customspectator view of the virtual reality experience for each spectator,whereby the view of that spectator can be correctly positioned andoriented in the scene in a way that corresponds to that audiencemember's actual position and orientation in the room.

BRIEF SUMMARY OF THE INVENTION

The present invention pertains to a system for a spectator to view avirtual reality of a performer. The system comprises a head mounteddisplay worn by the performer which produces the virtual reality for theperformer to see while wearing the head mounted display. The headmounted display having a head mounted computer which produces thevirtual reality on the head mounted display and has a transmitter forsending output signals corresponding to the virtual reality appearing onthe head mounted display. The system comprises a network which receivesthe output signals from the head mounted display corresponding to thevirtual reality being viewed by the performer. The network incommunication with the head mounted display. The system comprises anaudience portion having an audience display for the spectator whichreceives input signals from the network corresponding to the virtualreality being viewed by the performer on the head mounted display anddisplays these input signals on the audience display for the spectatorto view. The audience portion having an audience computer which displaysthe input signals on the audience display and an audience receiver whichreceives the input signals from the network. The audience portion incommunication with the network.

The present invention pertains to a method for a spectator to view avirtual reality of a performer. The method comprises the steps oflinking a telecommunications network to a smart phone of the spectatorand a head up display worn by a performer that displays the virtualreality. There is the step of viewing on an audience display of thesmart phone by the spectator the virtual reality that appears on thehead mounted display.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

In the accompanying drawings, the preferred embodiment of the inventionand preferred methods of practicing the invention are illustrated inwhich:

FIG. 1 is a block diagram of the present invention.

FIG. 2 is a representation of a spectator and a performer with a headmounted display having a virtual reality.

DETAILED DESCRIPTION OF THE INVENTION

Referring now to the drawings wherein like reference numerals refer tosimilar or identical parts throughout the several views, and morespecifically to FIG. 1 thereof, there is shown a system 10 for aspectator 12 to view a virtual reality 14 of a performer 16. The system10 comprises a head mounted display 18 worn by the performer 16 whichproduces the virtual reality 14 for the performer 16 to see whilewearing the head mounted display 18. The head mounted display 18 havinga head mounted computer 20 which produces the virtual reality 14 on thehead up display and has a head mounted transmitter 22 for sending outputsignals corresponding to the virtual reality 14 appearing on the headmounted display 18. The system 10 comprises a network 24 which receivesthe output signals from the head mounted display 18 corresponding to thevirtual reality 14 being viewed by the performer 16. The network 24 incommunication with the head mounted display 18. The system 10 comprisesan audience portion 26 having an audience display 28 for the spectator12 which receives input signals from the network 24 corresponding to thevirtual reality 14 being viewed by the performer 16 on the head mounteddisplay 18 and displays these input signals on the audience display 28for the spectator 12 to view. The audience portion 26 having an audiencecomputer 30 which displays the input signals on the audience display 28and an audience receiver 32 which receives the input signals from thenetwork 24. The audience portion 26 in communication with the network24.

The audience portion 26 may include a mobile smart phone 34. The virtualreality 14 displayed on the audience display 28 of the smart phone 34may correspond to an actual position and orientation of the spectator 12in the room in which the performer 16 having the head mounted display 18and the spectator 12 are located. The system 10 may include multiplespectators 12 each having their own smart phones which display on theiraudience display 28 of their smart phones the virtual reality 14 whichcorresponds to their actual position and orientation in the room.

The network 24 may include a Wi-Fi server that produces position andorientation updates of the performer 16. The smart phone 34 of thespectator 12 may compute a position and orientation of the smart phone34 with respect to the room and then concatenate's transformations ofthe performer 16 and the smart phone 34, using matrix multiplication, tocompute a time varying position and orientation of the room andtherefore of the performer 16 having the head mounted display 18 withrespect to the smart phone 34 has a smart phone 34 moves and rotates inspace through time. The performer 16 may make changes to the virtualreality 14. The performer 16 may wear a performance capture suit to makechanges to the virtual reality 14.

The smart phone 34 may have touch controls and the spectator 12 usestouch controls on the smart phone 34 to change a view of the virtualreality 14 appearing on the audience display 28 which differs from theposition and orientation of the smart phone 34 in the room. Thespectator 12 may use the touch controls on the smart phone 34 to makechanges to the virtual reality 14. A video flow captured from a cameraof the smart phone 34 may be used as an environment layer in the virtualreality 14.

The present invention pertains to a method for a spectator 12 to view avirtual reality 14 of a performer 16. The method comprises the steps oflinking a telecommunications network 24 to a smart phone 34 of thespectator 12 and a head up display worn by a performer 16 that displaysthe virtual reality 14. There is the step of viewing on an audiencedisplay 28 of the smart phone 34 by the spectator 12 the virtual reality14 that appears on the head mounted display 18.

The audience portion 26 may include a mobile smart phone 34. There maybe the step of displaying the virtual reality 14 on the audience display28 of the smart phone 34 corresponding to an actual position andorientation of the spectator 12 in the room in which the performer 16having the head mounted display 18 and the spectator 12 are located.

In the operation of the invention and with reference to FIG. 2:

1—VR users as performers: Each person wearing a VR device, such as anHMD, which can in one embodiment be a VIVE Virtual Reality system and anMSI computer worn in a backpack, and in another embodiment can be aperson wearing a performance capture suit, using one of various standardmethods for performance capture, while wearing a head-mounted VR device,such as a GearVR running a real-time simulation using standard softwaresuch as the Unity Game Engine, which is continually receiving positionalupdates from a Wifi server, can act as a performer 16 in the room.

2—People with smartphones as spectators: Each person holding asmartphone can experience the performance as a spectator 12, seeing intoa shared virtual world through the screen of their phone in a way thatcorresponds to their actual position and orientation in the room. Theposition and orientation of each VR actor is provided to a servercomputer over the shared wireless network 24 using a standard dataprotocol for describing position and orientation such as is providedwith the Vive virtual reality system or by the Optitrack motion capturesystem. This provides a base position and orientation that isindependent of the position of any given observer. Because the softwareon the phone is able to compute the position and orientation of thephone itself with respect to the room, the software on the phone is thenable to concatenate the transformations of the actor and the phoneitself, using standard Matrix multiplication, to compute the timevarying position and orientation of the room and therefore of the VRactor participant with respect to the phone, as the phone moves androtates in space through time.

3—Wireless local network 24, such as Wifi: The positional data from allperformers is sent via the wireless local network 24 to each spectator'ssmartphone.

People with smartphones are inside-out tracked, moving around, changeviews: Each smartphone is running software that does inside out trackingto determine the current position and orientation of that smartphone.

The inside out tracking, which is known in the art to identify thelocation and orientation of the phone, allows the phone to computewithout the use of any external markers in the room the time varyingposition and orientation of the phone as it moves through the room.

In one embodiment, this tracking is affected by sensor fusion between(a) the Inertial Measurement Unit (IMU) in the smartphone to performrapid updates of position and orientation and (b) a standard SIFTalgorithm to locate features in the room and use those features todetermine a ground truth of position and orientation.

The sensor Fusion algorithm breaks up the problem of tracking theposition and orientation of the phone within the room into two subproblems. One sub problem is determination of the absolute position andorientation of the phone in the room. This can be implemented by astandard Vision algorithm such as SIFT, which looks at ambient visualmarkers such as corner points of objects and corners of the room anduses the variations in visual position of those markers over time tocompute a Time varying Matrix which gives the movement of the phonethrough the room. A SIFT algorithm may not be sufficient forhigh-quality performance because it is limited by the frame rate andresolution of the phone's camera. For this reason, Sensor Fusion isemployed with the solid state gyroscope which is part of the IMU in thephone. The IMU gyroscope is able to operate at a much higher frame rateand therefore is able to fill in the short intervals of time betweencamera measurements by sending small time varying changes in orientationand adding those as offsets to the orientation computed by the SIFTalgorithm.

Hover mode, virtual buttons to moving around: Optionally the spectator12 can use touch controls on their phone's touch sensitive screen tomove their view into the virtual version of the shared world to aposition and orientation that differs from their actual position andorientation in the room.

Interaction among people with mobiles and VR HMDs: Each spectator 12 canoptionally also use the touch controls on their phone's touch sensitivescreen to communicate changes in the shared virtual scene, such as theposition of virtual objects or interactively animated creatures, throughthe wireless local network 24, so that those changes are seen by theperformers in the VR space, as well as by other spectators.

Structure of the System 10

In the system 10, there are two groups of users. One group of users canbe called “VR users”, who will use a Head-mounted display such as an HTCVive, Oculus or other kind of similar device, which can track both theuser's position and orientation. The other group can be called “mobiledevice” users, who will use mobile phones or tablets as their interfaceto the experience. Both types of users share the same virtual world viaa multiplayer based application on their devices and are also physicallycollocated during the experience.

On the software side, there is a pipeline for one or more of the usersof either or both of these two types of these users to share the sameexperience at the same time. Generally speaking, the applicationsoftware is generated by commonly available runtime software enginessuch as (but not limited to) Unity, Unreal and WebGL (Unity is a 3D gameengine to build up 3D games and experience, which is under the UnityCompany[website link: https://unity3d.com/]; Unreal is something similarto Unity, [website link:https://www.unrealengine.com/en-US/what-is-unreal-engine-4]); WebGL is aroyalty-free web standard for a low-level 3D graphics API based onOpenGL ES and used for browser based rendering [wikipedia:https://en.wikipedia.org/wiki/WebGL]. These tools have the capability tosupport 3D rendering on VR devices and mobile devices. Also, theseapplications need to be set up using commonly available multiplayerapplication logic to link the users via a shared network 24. Detailsbeyond these common multiplayer applications which are added to theseexisting multiplayer applications are:

Tracking Solutions

Since all users are sharing both the physical and virtual world, it isnecessary to know where the users are located. For the VR users, it iscommon in the industry to provide at least head position and orientationtracking for them. For this purpose, different products take usage ofmultiple ways to do so. For HTC Vive [https://www.vive.com/us/] uses theLighthouse system, whereas Oculus[https://www.oculus.com/] usesfixed-position cameras that track infrared markers on the headset. Thereare also inside-out tracking based on cameras and Inertial measurementunit, such as is used by the ViveFocus[https://www.vive.com/cn/product/vive-focus-en/]. Those trackingtechnologies are applied to the mobile devices. But in general, thetracking solutions on mobile smartphones will be more based oninside-out approaches since the mobile phones already have built-incameras and imu. Those solutions can currently be provided via thirdparty libraries such as ARKit [ARkit is a common library for inside-outtracking in mobile phone, under IOS, owned by Apple, link:https://developer.apple.com/arkit/], ARCore [ARCore is a common libraryfor inside-out tracking in mobile phone, under Android, owned by Googlelink:https://developers.google.com/ar/],ARToolkits[https://artoolkit.org/, similar to the above things butopen-source], Vuforia[https://www.vuforia.com/, similar to the resourcesabove but not open source], or implemented based on open-sourcesolutions such as VINs[https://github.com/HKUST-Aerial-Robotics/VINS-Mono].

Network 24 Sync-Up, Events

Since the experience is multiplayer based, it will require a network 24solution to sync-up all users on the same page. It is common in theindustry to sync-up all users into the same shared virtual world duringthe multiplayer experience. The features that are added to existingsoftware are:

Sync up physical locations based on the VR users and Mobile userstogether;

Sync up touch-events from Mobile users and controller events from VRusers together;

Sync up necessary virtual events (i.e., changes of Virtual objects,variations in the virtual avatars which presents other users).

The set-up of the network 24 can be a server-client based set-up whichincludes a server run in the back end to receive and broadcast to allclients, or can be some other method used in existing multiplayer games(such as point to point mesh networks).

Coordinator Synchronization

To generate the co-located experience among the VR users and Mobileusers, it is necessary to sync up their coordinate systems with theirphysical locations and virtual worlds. Details of the methods for thisare described below as follows:

Set up an origin point in the physical space as the (0,0,0) point ineach user's view of the virtual world. Since the physical position foreach device based on the tracking system 10 is known, this origin pointin multiple devices is synchronized as the common origin point.

Since the units in the real world are in length units like meters andthe units in the virtual world are in dimensionless numbers, to matchthose units it is necessary to set a ratio for conversion. Here theconversion constant “a” is defined, to match a meter length in thephysical world to a dimensionless unit length in the virtual world.

If the user is in the world coordinates as (x(meter), y(meter),z(meter)), his represented position in the virtual world will be (ax,ay, az). Then, as a matter of convention, taking a=1, keeps 1 meter=1units. “a” can also be defined as any real number. The orientation ofthe user will be in the same units in the physical world and the virtualworld.

Based on the ratio “a” defined above and the tracking data, now all theusers' projected positions in the virtual world are known. For one user,the avatars in other users in their proper positions (which is notnecessary) can be rendered, and that user's virtual camera view can beset-up into the shared world based on that user's location andorientation.

To set-up the Field-Of-View of the virtual camera, each VR user's cameraFOV is matched to their HMD's target FOV. However, if the Mobile usersare just rendering the virtual world, the FOV for them can be arbitrary.For best practice, a FOV of approximately 100 degrees will seem naturalfor the experience, but that limitation is not necessary. As an example,the Holodoodle project as a case can be used.

In the Holodoodle set-up, there is a virtual party in the virtual world.The VR users will see the whole set-up of the party: Everyone willbecome robotic-look like avatars, on a stage with visual effectsdecorated elements. They can use their hands to interact with theenvironment, i.e. open up the virtual light on the stage or launch afirework with their hands and the controller. For the VR users, theexperience is totally immersive.

For the mobile phone users, instead of experience the immersive versionlike the VR users, they will see everything on their mobile device. Thescreen will look like a filter to show the “virtual world” to them. Thatis to say: If they are using the mobile device targeting to the VRusers, they will see the robotic style avatars like them; if theyholding the mobile device to walk around, their perspective view willchange, just like they are walking inside the VR world. Things will besimilar to the Mobile users, as they turn on the camera on the mobilephone to walk in the real world, the difference will just be that allthe things they are experiencing will be in the virtual world. Theirinteraction towards the virtual world will be mostly achieved bytouching the screen, like if they want to launch the firework, afterthey selected the commands on the screen, from the touch point a raycasttowards the virtual world will be introduced, and then trigger theevents on the virtual worlds. The Mobile Users will also have avatars inthe virtual worlds, and can also be seen by VR users and Mobile Users.

Extension Cases

Here a description has been made of a system 10 having “VR” users and“mobile device users. This description can be extended. The “VR users”can be extended to be “a user with a head mounted display 18 which istracked in its position and orientation.” In this case, if the users areusing some mixed reality devices such as Hololens Hololens[Owned byMicrosoft, a mixed reality device which allow users to see theholographics in the 3D space and under inside-out tracking. Link here:https://www.microsoft.com/en-us/hololens] and Meta[similar to Hololens,but owned by Meta company. Link here: http://www.metavision.com/], theycan still be defined as “VR Users,” since they are using positionaltrackable HMD-based devices. The “Mobile device Users” do not need to belimited to just use a mobile phone, a device such as a tablet withtracking capability can also be used.

The position tracking does not always need to be extremely accurate. Forexample, if the mobile users are placed in a theater environment, it isnot necessary to track the mobile users' position if they are sitting ina seat. Since the location of each seat is known already, their positioncan be set based on their seat position. The same logic can be appliedto the VR users if they are only using DOF devices with orientation.[This means the VR devices don't have the positional tracking itself,but just have the orientation tracking] If their locations can beestimated due to some physical set-up such as their seat location, thenthis position information can just be used.

For the mobile phone users, partially rendering of virtual world objectsis also available. Based on how the experience is designed, the mobilephone experience can also be made as “Mixed Reality”, which in generalis a video see-through experience to merge the VR experience and thereal world together.

In order to achieve those “Mixed Reality” experiences, the change of themobile users' application will be:

1. Using the camera on the mobile phone to capture the video flow of thereal world and set up the video flow captured from the camera as the“environment layer” in the applications.

2. Adding on the essential virtual objects and the virtual avatars ontop of the environment layers, rather than rendering all the componentsin the VR world, in order to make the video see-through effects act likea “Mixed Reality” experience. Methods (On the mobile side, no differenceon the VR side or on the server side):

In the application side, using the camera as an input, using the cameracaptured frames as a texture, rendering this layer as the backgroundlayer on the applications.

On the development stage, all the tags on different objects, i.e.environments, backgound, interactable objects, avatars, so on will needto be made.

Based on the tags that were presetted, on the Mixed Reality modes, noteverything from the virtual world will be rendered; the wholeenvironment, the background of the VR world won't be rendered, but theinteractable objects and the avatars will be rendered on the correctprojection.

In order to match the camera on the mobile phone and the virtual camera,the virtual cameras on the mobile users' side need to be set as the sameField-Of-View as the camera on the mobile phone. The methods are asfollows:

Obtaining the Field-Of-View from the camera is not difficult. Most ofthe cameras on current phones will provide the FOV in the systemparameters. (If not provided, measure this with modern optical methodsis straight forward, which is not essential in this loop.)

In the Mobile users, a virtual camera needs to be set up in the gamescene to render everything. And the FOV is one of the importantparameters on this class. Simply set the FOV parameters to be totallythe same as the FOV of the physical device.

If the mobile phone has the ability to obtain depth information from theenvironment, it can be chosen to apply occlusion from the video layersto the virtual objects. In which, the methods will be:

Based on the depth information we've got from the depth sensors in themobile device, determine the relationship of if the physical worldobjects is blocking the virtual world objects.

If the virtual world objects are blocked, don't render the occlusionpart based on the depth information.

Here is an example as for the Holodoodle:

For the VR users, they experience the same thing.

But for the Mobile Users, once they are taking usage of the “MixedReality” experience, on the mobile device side, they won't render thevirtual stage, but just see the real world based on their camera on thephone. The avatars and virtual objects will still be rendered and underthe same perspective of views, and added on the real world layers.

3. If the mobile phone has the ability to obtain depth information fromthe environment, occlusion can be applied from the video layers to thevirtual objects.

STEP BY STEP OPERATION BY USER:

1. In an optional initial set-up stage, the location and orientation ofeach spectator 12 can be aligned within the room.

2. The smartphone of each spectator 12 is linked to the same Wifinetwork 24.

3. An application that performs the position and orientation tracking,as well as showing the shared virtual world on the smartphone screen, isdownloaded through the wireless network 24 onto the spectator's phone.Optionally, this downloading can be done beforehand in any location.

4. Each spectator 12 runs the application that supports the sharedvirtual experience.

EXAMPLE USE OF INVENTION

One example project that would be able to benefit from this invention isa shared “drawing in the air” application, in which each VR performer isdrawing a virtual 3D drawing in the air, as has been previouslydemonstrated by Google's TiltBrush VR application. In this example use,all spectators 12 would be able to walk around the room and see these 3Ddrawings as though they are physical objects in the room, and as thoughthe spectator 12 is viewing that virtual physical object in its properposition and orientation, as seen from the spectator's location.

Although the invention has been described in detail in the foregoingembodiments for the purpose of illustration, it is to be understood thatsuch detail is solely for that purpose and that variations can be madetherein by those skilled in the art without departing from the spiritand scope of the invention except as it may be described by thefollowing claims.

1. A system for a spectator to view a virtual reality of a performercomprising: a head mounted display worn by the performer which producesthe virtual reality for the performer to see while wearing the headmounted display, the head mounted display having a head mounted computerwhich produces the virtual reality on the head mounted display and has ahead mounted transmitter for sending output signals corresponding to thevirtual reality appearing on the head mounted display; a network whichreceives the output signals from the head mounted display correspondingto the virtual reality being viewed by the performer, the network incommunication with the head mounted display; and an audience portionhaving an audience display for the spectator which receives inputsignals from the network corresponding to the virtual reality beingviewed by the performer on the head mounted display and displays theseinput signals on the audience display for the spectator to view, theaudience portion having an audience computer which displays the inputsignals on the audience display and an audience receiver which receivesthe input signals from the network, the audience portion incommunication with the network.
 2. The system of claim 1 where theaudience portion includes a mobile smart phone.
 3. The system of claim 2wherein the virtual reality displayed on the audience display of thesmart phone corresponds to an actual position and orientation of thespectator in the room in which the performer having the head mounteddisplay and the spectator are located.
 4. The system of claim 3including multiple spectators each having their own smart phones whichdisplay on their audience display of their smart phones the virtualreality which corresponds to their actual position and orientation inthe room.
 5. The system of claim 4 wherein the network includes a Wi-Fiserver that produces position and orientation updates of the performer.6. The system of claim 5 wherein the smart phone of the spectatorcomputes a position and orientation of the smart phone with respect tothe room and then concatenate's transformations of the performer and thesmart phone, using matrix multiplication, to compute a time varyingposition and orientation of the room and therefore of the performerhaving the head mounted display with respect to the smart phone as asmart phone moves and rotates in space through time.
 7. The system ofclaim 6 wherein the performer makes changes to the virtual reality. 8.The system of claim 7 wherein the performer wears a performance capturesuit to make changes to the virtual reality.
 9. The system of claim 8wherein the smart phone has touch controls and the spectator uses touchcontrols on the smart phone to change a view of the virtual realityappearing on the audience display which differs from the position andorientation of the smart phone in the room.
 10. The system of claim 9wherein the spectator uses the touch controls on the smart phone to makechanges to the virtual reality.
 11. The system of claim 10 wherein avideo flow captured from a camera of the smart phone is used as anenvironment layer in the virtual reality.
 12. A method for a spectatorto view a virtual reality of a performer comprising the steps of:linking a telecommunications network to a smart phone of the spectatorand a head mounted display worn by a performer that displays the virtualreality; viewing on an audience display of the smart phone by thespectator the virtual reality that appears on the head mounted display.13. The method of claim 12 where the audience portion includes a mobilesmart phone.
 14. The method of claim 13 including the step of displayingthe virtual reality on the audience display of the smart phonecorresponding to an actual position and orientation of the spectator inthe room in which the performer having the head mounted display and thespectator are located.